You have a large graph that models some causal process mapping inputs to outputs. The graph models a diverse set of processes and it is strangely good at modeling them. You suspect that the graph is composed of subgraphs that are causally relevant to I/O and performing some processes (deeper and ultimately more useful computation with more units), and [ablating](https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=fh-HJyz1CgUVrXuoiban6bYx) certain subgraphs would cause this graph to model some processes worse. However, this graph was modeled using lots of different data points that you could not hope to scrutinize (it was trained on [The Pile](https://pile.eleuther.ai/)), so there are many cases where you can't just curate a list of datapoints and find the subgraph(s) that were used ([this doesn’t always work either!](https://www.lesswrong.com/posts/3ecs6duLmTfyra3Gp/some-lessons-learned-from-studying-indirect-object?commentId=4yBbKLoYxvr3KNohz))

So you devise some algorithms to “discover” subgraphs without explicit curation or use of datapoints. And because you entrust this graph to economically useful things, you want algorithms ensuring discovery of subgraphs consisting of o.o.d. “features” (taking deliberate steps to learn about units the model uses, rather than on some dataset, and doing so in a more adversarial manner)

What’s a quantitative way to find which algorithms yield more bits per explanation (in this case per circuit), while also making compute tradeoffs? If you can identify which algorithms yield more bits, you can reliably update more towards approximate mechanistic understanding of the target model[^1]. [Circuit robustness](https://www.lesswrong.com/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing) and [superposition interference](https://www.lesswrong.com/posts/JawMNDrcRsRrDcXFd/how-much-superposition-is-there) seem like decent quantitative metrics, and because understanding in-distribution behavior is typically overdetermined relative to o.o.d. behaviors, another factor should be distributional coverage. Conveniently enough, these are also individual tradeoffs

- Algorithms that incentivize sparsity or [minimum description lengths](https://www.lesswrong.com/posts/G2oyFQFTE5eGEas6m/interpretability-as-compression-reconsidering-sae) are not necessarily useful here. It takes more effort to discover subgraphs that are o.o.d., and with typical unsupervised methods it would take especially more effort for subgraphs consisting of effectively o.o.d. units (as a probable consequence of incentivizing these attributes). These algorithms should generally yield less o.o.d. bits
	- If you extrapolate out to [continual learning](https://www.lesswrong.com/posts/j2zBqt7AksoEoHXNp/how-might-continual-learning-affect-safety-and-alignment) or [deceptive alignment](https://www.lesswrong.com/posts/zthDPAjh9w6Ytbeks/deceptive-alignment) these approaches seem even less compelling (retraining seems impractical); even now, such methods likely cannot identify features for robust backdoors without distributional coverage
	- However, discovering circuits consisting of true features disentangled from superposition should yield more robustness/non-superposition-interference bits
- More o.o.d. coverage is desired here[^2], and this is what non-MDL algorithms could be better at, but it is inherently harder to make observations about robustness, and examples generated with model assistance or by methods akin to [feature](https://distill.pub/2017/feature-visualization) [visualization](https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation) are overly adversarial/unrealistic and don't help much

It mostly seems that the weakest link is distributional coverage, and also harder to measure, while superposition non-interference and robustness are often correlated; some methods that seek o.o.d. coverage tend towards gathering proof-of-existence type evidence (while seeking apparent generalization). Clear things to try with this in mind include stuff along the lines of integrating parts that improve metrics, like, how well do SAE/transcoder features approximate o.o.d.-seeming circuits discovered with other methods, or vice-versa (features recovered by other methods used in place of e.g. [greedy search](https://www.lesswrong.com/posts/YmkjnWtZGLbHRbzrP/transcoders-enable-fine-grained-interpretable-circuit#A_more_detailed_description_of_the_computational_graph_algorithm) methods)? What various methods exist to describe o.o.d. circuits that cannot be accurately explained by substituting for SAE/transcoder features or feature visualization? Are there any concrete models of what to expect of truly o.o.d. circuitry? Is there ways to adapt various algorithms into [lens-y](https://arxiv.org/abs/2303.08112) tools?

It also doesn't help that causal scrubbing/resample-ablation is supposed to find loss recovered from hypotheses on some distribution, and this approach to verifying hypotheses happens to be far more principled than say, zero-ablation, and even for zero-ablation, it is still quite useful to have some prior expectation of how a model behavior changes. Another thing to determine from here, is how to find a distribution, which again I think comparing with algorithms incentivizing MDL/sparsity is quite useful

## feature bases may be intractable for measuring robustness 

It seems incredibly likely that transformer components are too coarse of a unit to explain loss well, while circuits constructed of features from some MDL/sparsity prior (or due to being data-dependent features) are more fine-grained but less likely to be robust to behaviors that are o.o.d. . Suppose though that we have a method that can model feature circuits from perturbations (being data-independent), and that those features are [privileged](https://www.lesswrong.com/posts/QrPxNDprJwXiE73Py/finding-features-in-transformers-contrastive-directions) by the model (more than other baselines)

Though you can likely avoid side effects of superposition by intervening from a more privileged basis, it seems at least somewhat harder in practice to validate hypotheses of similar scope as component-level hypotheses (stating that a set of feature circuits are wholly responsible for mediating some behavior)

- It should ~always be more likely that more than one pair of input/output feature directions are responsible for mediating some behavior, than for a component-level subgraph
	- This seems basically fixable by identifying as many possible ‘related’ I/O pairs, and doing the equivalent of a pullback operation 

[^1]: the outcome of this looks like a benchmark, since all sub metrics here don't seem usefully robust under optimization 
[^2]: o.o.d. relative to SAE-like bases